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,^ ■ Abstract 

For infinitely large sparse networks of spiking neurons mean field theory shows that a balanced 
state of highly irregular activity arises under various conditions. Here we analytically investigate 
the microscopic irregular dynamics in finite networks of arbitrary connectivity, keeping track of all 
individual spike times. For delayed, purely inhibitory interactions we demonstrate that the irregular 
dynamics is not chaotic but rather stable and convergent towards periodic orbits. Moreover, every 



^ I generic periodic orbit of these dynamical systems is stable. These results highlight that chaotic and 

'NT . 

' stable dynamics are equally capable of generating irregular activity. 

cn 

lO ■ PACS numbers: 05.45.-a, 89.75.-k, 87.10. +e 

o ■ 
l> 
o 

> 



Highly irregular dynamics is a prominent feature of multi-dimensional complex systems 
and often attributed to chaos . Networks of spiking neurons, which interact by sending 
and receiving electricalpulses (spikes) exhibit very irregular dynamics for a wide range of 
condition. For insta..ce, ne.wor,.s of .parse random connectivity nray display 

a balanced state [3| in which excitatory (positive) and inhibitory (negative) inputs to each 
neuron balance on average and only the fluctuations create spikes at irregular points in time 
[3|, y|. Their dynamics resemble random (e.g. Poisson) processes with low correlations across 
different neurons. Mean field theory [J| shows that such irregular balanced activity occurs 
in networks with excitatory and inhibitory recurrent feedback as well as in networks that 
receive external excitatory inputs and exhibit recurrent inhibition only [J, 15|. Interestingly, 
for inhibitory networks recent work [l^ suggests that this irregular dynamics might be 
stable, at least in globally coupled networks and in networks without delay. In particular, 
Zillmer et al. jl^ numerically measured a negative maximal Lyapunov exponent of the 
irregular, seemingly chaotic spiking dynamics. 

In this Letter, we analytically investigate the microscopic dynamics in finite networks of 



For networks of arbitrary connec- 



spiking neurons with delayed inhibitory interactions 
tivity we show that almost all spike sequences generated by the network - trajectories of the 
dynamical system - are stable, i.e. sufficiently small perturbations of such trajectories do 
not grow with time. In particular, every generic periodic orbit in such inhibitory networks 
is stable. Moreover, the analysis indicates that all attractors in such networks are stable 
periodic orbits, which, however, may only be reached after very long and irregular, but also 
stable transients. 

Consider neurons that interact on a directed network by sending and receiving spikes. 
The subthreshold dynamics of the neurons' membrane potentials Vi(t) at time t are given 
by 

jv. = m) + E E ^^.^-^ - - ^..) (1) 

j=l fcGZ 

where a smooth function fi specifies the internal dynamics, j < is the inhibitory coupling 
strength and Tjj > the delay time of a synaptic interaction from neuron j to neuron i, 
and t^y. determines the time of the /cth spike sent by neuron j. Here 5{.) is the Dirac delta- 
function. If a neuron j reaches the threshold potential, Vj{t~) = Vej, it generates a spike 
at t =: t^ji^ for some k and is reset, Vj{t^j i^) = 0. We here require that the fj for all j satisfy 



fjiVj) > and f/iVj) < for all Vj < Vej such that in isolation each neuron exhibits 
oscillatory dynamics. 



1J| by a phase-like variable (f)j{t) G {—oo,(j)Qj] 



The neurons are equivalently described 
satisfying the linear differential equation 

d(f)j/dt = 1 (2) 

at all non-event times. Upon reaching a phase threshold (pj(t^j^i^~) = 0e,j, this phase is 
reset, (pjifjk) '■— a spike is generated. After a delay time Tjj that spike is received by 
postsynaptic neuron i of neuron j and its phase changes according to 

Hth + n,) = {^^ (fe + ^..)~)) • (3) 

This interaction is mediated by the transfer function 

Hf{<P) = Ur^[U,{<j>)^e] (4) 
where Ui{t) is the free (all Sij = 0) solution of ^ thr oug h the initial condition Ui{0) = 0, 



yielding U- > and U-' < 0, and 0e,j = U^^^Vqj), cf. 13||. For instance, for standard leaky 
integrate-and-fire neurons, where fi{V) = h — 'jiV with time scale 7^"^ > and equilibrium 
potential > Ve^j, we have f/i(0) = 7~^/j(l — exp(— 7^0)). Whereas the analysis below is 
valid for general Ui{(f)), all numerical simulations are presented for integrate-and-fire neurons. 
In the following we consider arbitrary generic spike sequences in which all neurons are active 
(i.e. there is a finite T > 0, arbitrarily large, such that in every time interval [t,t + T), 
t G M, every neuron fires at least once) and no two events occur at the same time. Sending 
and receiving of spikes are the only nonlinear events occurring in these systems. 

Highly irregular spiking sequences (cf. Fig[Th) constitute a typical form of activity in these 
networks, suggesting that the underlying dynamics may be chaotic. However, as we show 
below for networks of arbitrary connectivity, this dynamics generically is stable. To show 
this, we first analytically study the exact microscopic dynamics of an original trajectory, as 
defined by the (arbitrarily irregular) sequence of events generated by the network, and a 
slight perturbation to it that keeps the order of events as in the original. 

The time of the nth event (sending and receiving) occurring in the entire network is 
denoted by t„ in the original sequence, and by t„ in the perturbed sequence. Here simulta- 
neous reception of the same spike at different neurons constitute one event. Analogously, at 
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Figure 1: (color). Stable irregular dynamics in a random network (A^ = 400, 7j~^/i = 4.0, Ve,i = 1, 
Tjj = 0.1, connection probability p = 0.2, ejj = —16). (a) The upper panel displays the spiking 
times (blue lines) of the first 50 neurons. The lower panel displays the membrane potential trajectory 
of neuron i = 1 (spikes of height Af/ = 2 added at firing times). The inset shows a histogram of the 
coefficients of variation CVj := Oij fii = (t: 
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over time, (b) Exponential decay of the maximal perturbation maxj 



averaged 
(blue dots) and the 



minimal margin k'"' (gray line) for one given microscopic dynamics, (c) Algebraic decay of the 



average minimal margin, (green dashed line, averaged over 250 random initial conditions) and 



the analytical prediction (no free fit parameter) of (black solid line). Additionally we show the 
minimal margin k^") for three exemplary initial conditions (gray lines), including that of panel b. 

a given time t, we denote the phases of neuron % by 0j(t) and 0i(t), respectively. Let 

AS"^ = (0i(^n) - 0.(tn)) - (tn - ^n) := ^ - 5^^"^ (5) 

denote the difference of the phases of neuron % between the two sequences after the rath and 
before the (n+ l)st event, corrected for the time shift ^t^"^ = tn — in between the sequences. 



If at the (n + l)st event some neuron j* sends a spike, the phase shifts 

^(n+l) _ ^(n) 

of all neurons i stay unchanged. Due to the linear phase dynamics (l2|) between the spikes, 
A^""*"^^ = A^*-* = —6t^^~^^'> also specifies the temporal shift of the (n + l)st event. At some 
ti = tn+i + Tj j. postsynaptic neuron i receives the spike sent by j*. The resulting phase 
shifts are given by 

aP =i/i:).(0, (tr)) - (0. (ir)) - (7) 

where = 4>i{ti^i) +tn+i + Tij* -ti^i and 0i (ti j = 0i(ti_i) + rij. are the 

phases just before spike reception. Using the identities (pi (tr) = (pi (ii j + Af + 
and = = -AJ.?^ we apply the mean value theorem in eq. (JT]) and obtain 

A«=c«-Af"n(l-cfVA5?^ (8) 

where c'p is given by the derivative cf^ = dH'i^^^.{(p)/d(p for some between (pi{tr) and 
0j {ti If neuron j* is not connected to neuron i, Si^j* = 0, the function ifi*^^. (0) = 
Hq\4') = is the identity map, such that the phase shift stays unchanged, Af^ = Af 
indeed cf^ = dH^^\(p) / d(p = 1 is independent of 0. If neuron j* is connected to i we find cf^ 
bounded by 

c^^:=unn{{H^'Jjm<c?<^M^ (9) 
where e^^ = max {^ij} and = min {£i,j}- The phase is confined to some finite 
interval G [0mn) 0mx] which depends on the network parameters. Given that dH£^\(f)) / d(p = 
U'i (0) /Ul {Uf^ [^i(0) + e:]) and using the monotonicity U- > and concavity < 0, we 
find Cjnn > and Cmx < 1, independent of the sequence and of the network realization 
(including its connectivity). Thus the phase shift after receiving a spike is a weighted 
average of earlier shifts. 

Consider that the perturbed sequence is created from the original one by perturbing 
the phases of all neurons and the sending times of all spikes sent but not received at time 
t = to. (We denote the maximum of these perturbations by Amx and the minimum byAL°^). 
After some finite time all spikes perturbed initially arrived and all perturbations to newly 
generated spike sending times stem from previous perturbations of some neurons' phases 



Given that the maximum phase shift cannot increase and the minimum not decrease 
according to ([8]) and Q, this implies Lyapunov stability of the dynamical trajectories. 



For strongly connected networks [15||, more involved graph theoretical arguments show 
that the initial perturbations actually decay exponentially such that the considered trajec- 
tories are asymptotically stable (cf. p, [l^). Briefly, following the propagation of a phase 
perturbation of one specific neuron Iq across the entire network, shows that after a finite 
number of := 2NM events all perturbations (of the phases of all neurons and of the 
sending times of spikes sent but not yet received) are bounded from above by 

A^fx^ < c*AfJ + (1 - c*)AW. (10) 

Here c* := (1 - Cmx)^(cmn)^^^^ < (1 - Cmx)(cmx) such that 3/4 < (1 - c*) < 1 and M 
depends on the spike sequence but is finite. Similarly we find a lower bound given by 
^inn > c*A\'^'^ + (1 — c*)A^n- The difference of the maximal and the minimal perturbation 
after K events is therefore given by 

I — mx mn — V / V mx mnJ V / 



This inequality STui implies that both sequences converge exponentially fast against each 
other. Thus all sequences considered are asymptotically stable for all strongly connected 
networks. 

A main condition for stability of trajectories was that the order of events stays the same in 
the perturbed and original trajectories. For arbitrary non-degenerate spike sequences, there 
is a non-zero perturbation size keeping the order unchanged in any finite time interval. 
However, the requirement of an unchanged event order yields more and more conditions 
over time such that the allowed size of a perturbation could decay more quickly with time 
than the actual perturbation. This will be excluded if the temporal margin /i*-"-* between two 
subsequent events stays larger than the dynamical perturbation for infinite time. Formally, 
after time t„ denote the kth potential future event time (of the original trajectory) that 
would arise if there were no future interactions by 6'.„ ^ , A; G N, and the temporal margin by 
yU*^"^ := 6n,2 — dnA- A Sufficiently small perturbation, satisfying A^x — A^n < Ai''"^ cannot 
change the order of the {n + l)st event. 

This directly implies that almost all periodic orbits (all those with non-degenerate event 
times tn) consisting of a finite number of P events are stable because there is a minimal 



margin 



k(^) := min u^"^) (12) 

ne{l,...,P} 



for every non-degenerate periodic pattern. 

To further analyze stability properties of irregular non-periodic spike sequences, we con- 
sider the minimal margin k^^^ over the first n events. Assuming that, along with the irregular 
dynamics, the temporal margins are generated by a Poisson point process with rate u, the 
distribution function of margins is given by P < /i) = l-e"^^. The probability that the 
minimal margin n^"^^ after n events is smaller or equal to /i is determined by the probabilities 
that not all individual margins /i*^"-' are larger than /i such that 

n 

P(k(") <fi) = l-Y[ > /i) = 1 - e-"""" (13) 

m=l 

with density p„(/i) := dP(K^'^^ < fi)/dfi = nv exp{—nuii). This implies an algebraic decay 
with the number n of events for the expected minimal margin 



= / fiPn{f^)dfi = {u ■ n)-^ (14) 
Jo 

that depends only on the event rate and is independent of the specific network parameters. 
Numerical simulations confirm this algebraic decay of the expected minimal margin 
with the number of network events n; an example is shown in Fig. [lb. Together with the 
exponential decay of dynamical perturbations (fTTl) this indicates that for a sufficiently small 
perturbation the order of events stays unchanged for infinite time. 

Interestingly, arbitrary irregular spike sequences necessarily converge to a periodic orbit 
after finite time (cf. Fig. [2]) because there is some finite number E such that (i) two sequences 
that share the same order of E events are equally ordered for all future events because 
any initial difference decays exponentially (fTTl) and (ii) there is only a finite number of 
orderings of events in a finite network such that a given sequence of length E repeats after 
finite time. Nevertheless, we find that the transient time until a periodic orbit is reached, 
rapidly increases with network size N and with the interaction strengths, in agreement with 

n n 

observations in ref. [10|; cf. also |16l |. 

In summary, we analytically accessed the microscopic dynamics of inhibitory networks 
of spiking neurons with arbitrarily complicated connectivity and delayed interactions. We 
showed that all generic trajectories are stable, even if they generate highly irregular dynam- 
ics. In particular, the results analytically confirm recent numerical findings [l^ that irregular 
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Figure 2: (color) Convergence towards a periodic orbit in a random network {N = 40, 7^ ^I- = 3.0, 



Ve,i = 1.0, Tjj = 0.1, p = 0.2, eij = —3.3). (a) The average minimal margin (as in Fig.dh) 
decays as a power-law (region A) and saturates after about 10^ events (region B) when the periodic 
orbit is reached. Inset: Margin /i^") (black) and minimal margin k^") (gray) for a trajectory started 
from one specific initial condition. The margin /x^"^ fluctuates strongly on the transient but is 
comparatively large and bounded after the sequence becomes periodic; thus the minimal margin 
K^") does not decrease further for future events n. (b), (c): Snapshots of irregular spike sequences 
(b) after n = 20.000 events on the transient and (c) after n = 10^ events on the periodic orbit. 



dynamics in spiking neural networks may exhibit stable behavior. Moreover, as for globally 
coupled systems vv^ithout delay our results shov^ that also networks with more complex 
structure and delayed interactions exhibit stable periodic orbits. However, highly irregular 
yet stable transient trajectories dominate the dynamics in all large and non-globally coupled 
networks, in stark contrast to the fast convergence to attractor dynamics found recently in 
globally coupled networks and also opposite to the long chaotic (and thus unstable) tran- 
sients in randomly diluted networks of excitatory neurons j^. In particular, in the class of 
systems presented here, almost every trajectory is stable, not only if it generates regular 
5|| but even if it generates very irregular dynamics. Curiously, the assumption of events 
generated by a maximally irregular (Poisson) random process led us to show stability of the 
deterministic trajectories. 

More generally, our results underline that multi-dimensional deterministic dynamics with 
statistical properties close to that of a random system need not be generated by deterministic 



chaos. As we have shown analytically for spiking neural networks with delayed interactions, 
dynamical irregularity may well be generated by stable trajectories, dominantly on a tran- 
sient not on an attractor. Future work needs to investigate closer the key consequences for 
systems in which irregular dynamics is stable. For instance stable irregular dynamics lifts the 
important practical constraint of long-term unpredictability that irregular dynamics bears 
if it is generated by chaos. Stable irregular dynamics, even in multi-dimensional systems, 
may well be predictable in practice. For the networks of spiking neurons studied above, this 
has the astounding consequence that the dynamics in only a small time window, even in the 
presence of some errors, defines the entire future of the highly irregular spiking dynamics. 

We thank Fred Wolf for helpful comments and the Federal Ministry of Education and 
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